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Models for Primary Care Residency Program Evaluation 

Abstract 

The purpose of this qualitative study was to develop a program evaluation template to be 
used for primary care residencies. Programs were reviewed by meta-evaluation, interviews, and 
a focus group. Survey components were: areas evaluated, evaluation beliefs and seven specific 
evaluation approaches. Emergent categories from interview and focus group content analysis 
were: assessment/evaluation areas; methodology; stakeholders; issues related to utility; issues 
related to feasibility; issues related to propriety; and needs which influence evaluation model 
choices and examples. Some of the evaluation approaches were not used by the programs. A 
final program evaluation template was developed to facilitate residency program evaluation. 

Purpose of the Study 

Many sponsoring institutions of residency programs have been facing or experiencing 
corporate downsizing in the last decade, while various stakeholders have raised questions of the 
effectiveness and efficiency of their residency programs. Program evaluation has become 
increasingly critical in determining the value and worth of such programs in corporate 
downsizing decisions. Program directors are being asked by their major stakeholders, funding 
sources, professional organizations, and beneficiaries to demonstrate not only the merit and 
worth of their programs, but also their progress in alleviating or solving social problems, and 
whether these programs are operating in a cost efficient manner as possible. Of equal importance 
is program accreditation. Many residency directors assess various parts of their program, mainly 
resident learning. This however, by itself, does not constitute program evaluation. A literature 
review found very few published studies of meta-evaluation, and/or programs that use several 
evaluation approaches to evaluate different aspects of the program. The main purpose of this 
study was to develop a theoretical evaluation model, or template, to be used for primary care 



residency programs (family practice, internal medicine, pediatrics, combined 
medicine/pediatrics) drawing from the evaluation literature, interviews, and based upon a meta- 
evaluation of two primary care residency programs’ evaluation systems. 

The following questions were addressed: 

I. How are primary care residency programs currently being evaluated? (approaches) 

A. Who are the primary stakeholders? 

B. What are the formally stated purposes of the evaluations? 

C. How has information from the evaluation process been utilized? 

II. What might an appropriate evaluation system/template look like? 

Theoretical Framework 

The conceptual framework of this study was utilization of several evaluation approaches 
(Deshler, in press) and meta-evaluation (The Joint Committee, 1994). The template was based 
upon seven evaluation approaches: the Connoisseurship Criticism Model (expertise oriented); 
the Attainment of Objectives Model and Goal-Free Evaluation Model (objectives oriented); the 
Differential Model and Experimental Model (decision-management oriented) and the Naturalistic 
and Participatory Evaluation Model (naturalistic and participant oriented). (See Table 1). Meta- 
evaluation is the practice of evaluating evaluation processes and products. The meta-evaluation 
concept provides that evaluations should be judged on their utility, practicality, ethics, and 
technical adequacy. 
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Table 1 



A Summary of Seven Program Evaluation Approaches 





Approach 


Orientation 


Focus 


Suqaested 
Methods of 
Gatherina Data 


Strenaths 


Limitations 


ConnoisseurshiD Criticism Model 
-Experts interview, analyze 
documents, and make judgments 
using their own judgment 
perspectives or those set as 
standards by the outside 
professional association that they 
are representing 


Expertise 

Oriented 


Inputs, activities, 
participation 


Document 
Analyses, 
Interviews & 
Questionnaires 


Emphasized expert 
judgment 

Standards used for 
evaluation 
Program goes 
through a self-study 
prior to site visit 


“Presumed" expertise 
of reviewers 
Evaluator judgments 
may be biased 
Goal side effects are 
not evaluated 


Attainment of Objectives Model - 
Evaluators measure the success 
of a program by measuring its 
outcomes against its own goals 
and objectives 


Objectives 

Oriented 


Compares 
inputs to 
outcomes 


Document 
Analyses, 
Observation, 
Interviews & 
Questionnaires 


Simple 

Measures Objectives 


Does not evaluate 
merit or worth of 
program and 
objectives 
Neglects context, 
unintended outcomes 


Goal-Free - Evaluators identify 
needs of learners and then 
compare these needs with what 
people are actually experiencing 
as a result of the program 


Objectives 

Oriented 


Individual, 
organizational, 
& community 
change 


Document 
Analyses, 
Observation, 
Interviews & 
Questionnaires 


Evaluates 
discrepancies 
between actual 
outcomes and 
objectives 


Time consuming 
Labor intensive, high 
cost approach 


Differential Model - This 
evaluation approach provides 
relevant information as a 
management tool to decision 
makers. Participation of 
stakeholders is central to the 
process 


Management 

Oriented 


Program 

planning 

through 

implementation; 

stakeholder 

participation 


Interviews & 

Questionnaires, 

Observation 


Evaluates each stage 
of a program 
Assesses program 
efficiency 
Provides useful 
information 


May result in costly 
and/or complex 
evaluations 


Experimental Model - This 
evaluation approach determines 
whether changes in program 
outcomes were due to the 
contributions of the program and 
not just to life’s experiences or 
other influences 


Management 

Oriented 


Causal 

relationships 

between 

activities and 

individual or 

organizational 

change 


Document 
Analyses, 
Experimental 
Design with 
Control Groups 


Determines impact 
and efficiency of a 
program 


Considered more as 
social research than 
evaluation 
Cost, educational 
liability of denying 
“treatment” 


Naturalistic Model - This 
evaluation approach's purpose is 
to understand how a program 
operates in its natural 
environment 


Naturalistic 
and Participant 
Oriented 


Activities, 
participation, & 
reactions as 
processes 


Observation, 
Interviews & 
Questionnaires, 
Document 
Analyses 


Participant value 
sensitive 
Feedback is in 
beneficiaries 
language 
Judgment shifted 
from evaluator to 
participants 
Contextually 
anchored' f 


If done in correctly , 
may be very 
subjective 
Time consuming 
Labor intensive, high 
cost approach 


Participatory Evaluation Model - 
Educators and learners 
themselves initiate a critical 
reflection process focused on 
their own activities by identifying 
anomalies and using critical 
reflection 


Naturalistic 
and Participant 
Oriented 


Anomalies in 

activities, 

emphasizes 

learner 

participation 


Interviews & 
Questionnaires, 
Document 
Analyses 


Participants leam 
more about the 
program 

Participants learn 
more about 
evaluation 
Participant “buy-in” 


Time consuming 
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A review of the literature documents that there have been no meta-evaluation studies of 



residency programs. The only two meta-evaluations found evaluated a specific rotation (Gauger, 
1985) and a faculty development program (Sheets, 1985). Many of the published residency 
program evaluation studies were focused upon assessment, or only utilized a single evaluation 
approach. The few articles written on program evaluation were theoretically based with little or 
no studies performed. The models reviewed in those articles included Stufflebeam’s CIPP 
model, and those which focused solely on objectives and outcomes. Table 2 is a schemata which 
shows the levels of evaluation, whether the focus is outcome and/or process, what the object of 
evaluation is, and how it relates to the evaluation models reviewed. 

Table 2 



Summary of Evaluation Levels, Focuses, Objects and Models 



Levels of 
Evaluation 


Focus of 
Evaluation 


Object of Evaluation 


Approach/Model 


Micro 


Intended 

outcomes 


Resident, faculty member 
Rotations, curriculum, 
educational experiences 


Attainment of Objectives 


Micro 


Unintended 

outcomes 


Resident, faculty member 
Rotations, curriculum, 
educational experiences 


Goal-Free Evaluation 
Participatory Evaluation 


Micro 


Development 


Resident, faculty member 


Differential Evaluation 




and decision 
making 


Rotations, curriculum, 
educational experiences 


• 


Micro 


Context 


Resident, faculty member 
Rotations, curriculum, 
educational experiences 


Naturalistic Evaluation 


tylacro 


Development 
and decision 
making 


Program 


Differential Evaluation 


Macro 


Intended 

outcomes 


Program 


Attainment of Objectives 
Experimental Evaluation 


Macro 


Context 


Program 


Naturalistic Evaluation 


Macro 


Unintended 

outcomes 


Program 


Participatory Evaluation 


Macro 


Inputs/processes 


Program 


Connoisseurship 
Criticism Model 


Evaluation 

Standards 


Evaluation 


Evaluation processes and 
products 


Meta-Evaluation 
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Methodology 



Interviews and focus groups were given to primary care program directors (n=4), medical 
educators (n=2), and an accreditation site surveyor (n=l). Reviewed for the literature search 
were two descriptors: internship and residency; and program evaluation. The two databases 
searched were the Educational Resources Information Center (ERIC), and Medline, a 
computerized division of the Medical Literature and Retrieval Systems (Medlars). In addition, 
related dissertations, program evaluation texts, evaluation journals, meta-evaluation literature and 
materials related to residency accreditation were examined. 

In the first part of this study, case study surveys were accomplished by separately 
interviewing one family practice and one combined internal medicine/pediatrics residency 
program director. Each director was administered the case study survey in person by the 
investigator. For example, each director was asked how program evaluation was accomplished. 
The program directors were asked for specific examples as to how attainment of objectives 
evaluation was achieved. If a particular approach was performed, they were next asked how the 
data from that evaluation approach were utilized. The process for utilization was also assessed. 

As the second step of this study, an evaluation matrix was developed based upon the 
evaluation literature reviewed. Based upon the program directors’ feedback from the case study 
surveys, the evaluation matrix, reviewed evaluation theory and metaevaluation of the programs, a 
template to evaluate primary care residency programs was developed. 

The third step of this study utilized a focus group. The focus group scheduled was 
comprised of two Ph D. medical educators, one internal medicine assistant residency director, 
and one pediatric residency director. During the focus group, the same questions were asked as 
in the previous case study surveys. In addition, the focus group was given the evaluation 
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template to review. Verbal feedback regarding the evaluation template was also obtained during 
the focus group. 

A second focus group was to be conducted with medical educators who review programs 
for the ACGME accreditation (AMA, 1 996). The same interview guidelines were to be used 
with this second group as was with the first focus group. Accreditation site surveyors live in 
many areas of the country. In fact, there are less than two dozen nation wide. The ACGME site 
surveyors fly to different programs most every week of the year. Because of this, it was 
impossible to schedule a focus group since they only convene very rarely at national 
headquarters. Because of these circumstances, in lieu of the second focus group, an individual 
interview was conducted using the same interview guidelines as the first focus group with an 
accreditation site surveyor who lived in a region near the researcher. 

The fourth step of this study was to refine the evaluation template based upon 
information received during the focus group and additional interview. A meta-evaluation was 
performed using data gathered during the family practice and combined medicine/pediatric 
interviews and document analysis. Based upon the meta-evaluation, feedback was presented to 
both residency directors individually about the current evaluation systems in place. For a 
member check, the two residency directors were also given a copy of the revised evaluation 
template for their comments on utility, feasibility, and any other pertinent information or 
feedback. Both residency directors were also asked to review the categories which emerged from 
the content analysis. 

The original contact with each participant was made by telephone to explain the project 
and to request an interview. No requests for interviews were refused. Telephone contact was 
again made to confirm the date, time, and place of the on-site interview. Relevant documents 
such as evaluation instruments, Internal Review committee reports, accreditation reports, and 



program brochures were examined to provide additional data. The data collection phase began in 

March of 1996 and concluded in August of 1996. A timeline of research is provided in Figure 1. 

1995 1996 

Months 3 6 9 12 3 6 9 12 

Activity 



1. Review of Literature * 

2. Development of Interview Guide 

3. Case Study Surveys 

4. Content Analysis 

5. Meta-evaluation of two programs 

6. Development/Refinement of Template 

7. Focus Group 

8. Accreditation Site Surveyor Interview 

9. Member Checks 

Figure 1. Timeline of research activities. 

Findings 

After performing meta-evaluation for the family practice and the combined 
medicine/pediatrics programs, the strengths and weaknesses of their evaluations can be 
summarized as follows: 

1 . For program evaluation, the main focuses of the program'evaluations were 
accreditation and program improvement. The approaches for gathering the information was 
verbalized, but not written. 

2. Both programs were thoroughly described in their respective RRC applications. 

3. Only the program director, some of the faculty members, sponsoring institution 
administration, and affiliated medical schools saw the evaluation results, although other 
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stakeholders such as residents and patients had a voice in the process. This is not to imply all 
stakeholders should have received the information in the same format and detail. 

4. The evaluations performed in the two residency programs were ethical and considered 
factors such as interpersonal skills and political ramifications. 

5. Although much information was gathered, the evaluation process was not always 
written down for reference. The evaluation process seems to have remained constant over the 
past several years. It was not a priority to evaluate the process, nor was there adequate time. If a 
new faculty member or administrator were hired by the program, they would have to review the 
resident evaluation folders, the faculty files, and the RRC documents to see the type of data 
gathered; where it was gathered; and how it was collated and analyzed. 

6. One major area that was not addressed by the evaluations performed by these two 
programs was unintended outcomes. Unintended outcomes were not specifically addressed 
unless they result in an incident which is subsequently reviewed. 

7. There was no reference to validity, reliability, and objectivity in any written materials 
on how the programs evaluate themselves. 

8. Considerable time was spent gathering data from residents on faculty or rotations that 
was not always collated or used. 

9. The written standards used by both programs were those provided by the RRC. These 
criteria are vague. The family practice program has additional criteria provided by their 
Residency Assistance Program (RAP) which was based upon the RRC but which was spelled out 
in slightly more detail and had higher standards than the RRC’s minimum expectations for 
accreditation. 

10. Judgments or recommendations concerning the entire program come mainly from the 
RRC and the institution’s Internal Review committee. The program directors normally made 



decisions concerning various components of the program, such as implementing changes in order 
to improve a specific rotation. 

11. The final reports in these two programs concerning program evaluation were written 
by people external to the program: the RRC; and the Internal Review committee. A complete 
report was viewed by top administration and the program director, while a summary was 
presented to other faculty and/or interested parties. The faculty members had opportunities to 
see the entire report if they desired. 

After speaking to all the study’s participants, program evaluation generally occurred only 
during accreditation and internal review processes. The programs tended to use fewer data 
sources when evaluating areas or people other than residents. All of the interview and focus 
group participants noted use of the Connoisseurship Criticism Model, the Attainment of 
Objectives Model, Differential Evaluation Model, and Participatory Evaluation Model when 
evaluating their programs. To a very limited extent, the Naturalistic Evaluation Model and Meta- 
evaluation were utilized. The Goal-Free Evaluation Model and Experimental Evaluation Model 
were not utilized by the residency programs. Two principal reasons were that the program 
directors were either unaware of them, or did not have the resources available to perform them. 
Although the program directors were not familiar with the Goal-Free Evaluation model, they 
were very intrigued by it. Major categories which emerged from the interview and focus group 
content analysis were: assessment/evaluation areas; methodology; stakeholders; issues related to 
utility; issues related to feasibility; issues related to propriety; and needs which influence 
evaluation model choices and examples (see Table 3). The major stakeholders which emerged 
from the content analysis were the program director, faculty and residents. Due to this finding, 
the final program evaluation template was divided into three categories: faculty, resident, and 



program. Based upon the meta-evaluation, interviews, and focus groups, and member checks a 



final template was developed. 

Table 3 

Categories Program Evaluation Derived from Content Analysis. 



Categories 

1.0 Assessment/Evaluation Areas 



2.0 Methodology 



3.0 Stakeholders 



4.0 Issues Related to Utility 

5.0 Issues Related to Feasiblity 

6.0 Issues Related to Propriety 

7.0 Needs which Influence Evaluation 
Model Choices and Examples 



0 




Category Components (# of responses) 

1 . 1 Curriculum (22) 

1 .2 Rotations/Special Courses (7) 

1.3 Budget (5) 

1 .4 Resident Applicants (5) 

1.5 Residents (18) 

1.6 Faculty (15) 

2.1 Competencies/ Accountabilities (5) 

2.2 Questionnaires/Surveys (3) 

2.3 Interviewing (1) 

2.4 Observation (4) 

2.5 Document Analysis (4) 

2.6 Peer Evaluation (2) 

2.7 Self-Evaluation (5) 

2.8 Standardized Exams (3) 

3.1 Faculty (3) 

3.2 Nurses (1) 

3.3 Residents (4) 

3.4 Patients (1) 

4.1 Report Timeliness/Dissemination (1) 

4.2 Values Identification (1) 

4.3 Evaluator Credibility (12) 

5.1 Practical Evaluation 
Procedures/Resources (9) 

5.2 Evaluation Cost Effectiveness (1) 

6.1 Ethics/Equity (2) 

7.1 Context Analysis (3) 

7.2 Defensible Information Sources (1) 

7.3 Impartial Reporting (1) 

7.4 Program Documentation (4) 

7.5 Purpose (9) 

7.6 Quantitative/Qualitative Analysis (6) 

7.7 Valid/Reliable Data (10) 

7.8 Expert Review (2) 

7.9 Objective/Outcome Assessment (5) 

7.10 Observational Evaluation(5) 

7. 1 1 Needs/Planning Assessment (7) 

7.12 Controlled Studies (4) 

7.13 Evidence-Based Reflection (8) 

7.14 Evaluation Systems Evaluation (12) 

10 12 



Revised Evaluation Template 



Program directors are likely to be unaware of the theoretical underpinnings of evaluation 
models and use techniques for pragmatic reasons and felt needs such as accreditation and 
graduation requirements. A template will give directors a systematic way of collecting relevant, 
useful data to make informed judgments about the merit or worth of their residency program. It 
will also afford them opportunities to gather data that might not have been taken into account 
before, or in ways that might not have been considered. By using an evaluation template which 
is theoretically based, and utilizing meta-evaluation, primary care residency directors can use the 
most appropriate evaluation approach for various situations at various points in time in their 



programs. 

The data provided by the case study surveys, meta-evaluation, content analysis and 
member checks give an insight into the complexities of program evaluation for residencies. 




Table 4 presents the revised evaluation template. The revisions were based upon comments 
made during the member checking process while participants reviewed the original template. 
These revisions and additions are indicated in the revised template in Table 4. 

Several areas of the revised evaluation template were not currently being evaluated. 
Resident evaluation areas included: identifying applicant demographics which predict those 
interested in primary care, and mock orals. Template items under faculty evaluation currently 
not being done included: feedback from nurses and patients, self-assessment, peer evaluation 
(including observation of teaching skills), auditing process for residents, and procedures taught. 
The third category, program evaluation, currently lacks overall procedures and diagnoses taught, 
external reviews by organizations or other program directors, and assessment of outcomes. All 
other items were performed in the two programs, with the exceptions of observation. The 
combined medicine/pediatrics program uses audiotapes, videotapes, and CEXs, while the family 
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practice program uses videotapes and OSCEs. The other areas mentioned above are items to be 
considered in the future for program evaluation. Using the categories which emerged from the 
content analysis, those items being done met those criteria identified in the categories such as 
propriety, utility and feasibility. Those not being done are the result of not having enough time, 
money or personnel. 

Table 4 

Revised Program Evaluation Template 



Faculty Evaluation (all information stored in evaluation folders for each individual - databases of aggregate data 
maintained for program evaluation) 

1 . Before entry into the program - faculty applicants 

a. Needs of program matched with faculty applicant qualifications 

b. Review letters of recommendation, board certification/recertification and licensure, comments made by 
interviewers. 

c. Obtain feedback from nurses and patients if available. a 

2. Monthly/Rotational 

a. Evaluations by residents/medical students 



3. Each Academic Year 

a. Self-assessment 

b. Peer evaluation 

c. Collated evaluations by residents/medical students reviewed 

d. Chart audits reviewed 

e. Auditing process for residents reviewed 1 

f. Procedures taught (procedures log) reviewed 

g. Patient satisfaction questionnaires reviewed 

h. Individual faculty goals and progress towards goals reviewed (faculty development) 

i. Videotapes and/or observations of teaching sessions reviewed 

j. State licensure and board certification maintained 

k. Incident reports reviewed 

l. Awards reviewed 

m. Research reviewed 

n. Publications reviewed 

o. Courses/lectures given reviewed 

p. Practice styles reviewed 

q. Membership and roles in regional, state, and national professional organizations 1 



3. Faculty termination 
a. Exit interview 



( table continues ) 
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Table 4 (continued) 



Resident Evaluation (all information stored in evaluation folders for each individual - databases of aggregate data 
maintained for program evaluation) 

1 . Before entry into the program - resident applicants 

a. Review letters of recommendation, dean’s letter, board scores, comments made by interviewers, licensure 
is attainable and granted 

b. When a resident applicant is given a patient case scenario, residency faculty assess each applicant’s ability 
to diagnose and manage based upon information given. 

c. Residents coming into a program with prior credit, need additional official documentation and need to 
apply in an ethical manner. 3 

d. Assess interpersonal skills. a 

e. Identify applicant demographics which would predict those individuals interested in primary care, and 
select applicants accordingly. a 

2. Each Academic Year 

a. 1st month - 

1 . Objective Structured Clinical Exams (OSCEs) or Clinical Exams (CEX) 

2. Self-Assessment - knowledge base, attitudes, skills 

b. Each month/rotation 

1. Evaluations by attendings/faculty 

2. Evaluations by senior/fellow residents 

c. Every six months 

1 . Chart audits review 

2. Procedure/diagnosis documentation reviewed 

3. Patient satisfaction questionnaires reviewed 

4. Rotation evaluations by faculty, staff and fellow residents reviewed 

5. Individual resident goals and progress towards goals reviewed 

6. Videotapes of patient encounters reviewed 

7. Audiotapes of patient encounters reviewed 

7. Practice styles reviewed 

8. Conference attendance and conference presentations reviewed 

d. Once a year - 

1. In-Training Exams 

2. Licensure maintained 

3. Self-Assessment 

e. As needed - 

1 . Incident reports reviewed 

2. Objective Structured Clinical Exams b 

3. Mock Orals a 

3. Graduation 

a. Exit interview 

b. Review specialty board results 

c. Each year, graduate surveys mailed one year after completion of residency program 



( table continues ) 
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Table 4 (continued) 

Program Evaluation Template 



Program Evaluation (databases of aggregate data is maintained for evaluation) 

1. Planning a new program 

a. Ad Hoc committee formed consisting of program stakeholders 

b. RRC mandates/guidelines reviewed 3 

2. Monthly/Rotational 

a. Evaluations by residents/medical students of rotation logistics (how the rotation occurs) 

b. Evaluations by residents/medical students of rotation content (what residents learned) 3 

3. Each Academic Year 

a. Review overall In-Training Examination results 

b. Collated evaluations of rotation logistics by residents/medical students reviewed 

c. Collated evaluations of rotation content by residents/medical students reviewed 3 

d. Chart audits/quality assurance reviewed 

e. Overall procedures taught/diagnoses managed reviewed 

f. Patient satisfaction questionnaires reviewed 

g. Overall faculty goals and progress towards goals reviewed/faculty development 

h. Descriptions of learning environments/processes written down 

i. Collated results of graduate surveys reviewed 

j. Collated results of resident applicants reviewed 

k. Incidents reviewed 6 

l. Conferences reviewed 

m. Program budget reviewed 

n Curriculum needs identified on a continual basis 3 

o. Existing curriculum reviewed on a continual basis 3 

p. Review evaluation reports from previous year(s) 

q. External reviews by organizations such as RAP 3 

r. Inviting well known program directors for a site visit 3 

s. Review organization/institution mission and vision statements and associated 
outcomes 3 

4. Accreditation 

a. Accreditation reports submitted when indicated 

b. Internal review conducted midway between accreditation time frames 



Note. a indicates revisions/additions from original template, b as incidents occur or to assess 
problem areas and implement changes (in addition, OSCEs may be used to assess 1st year 
residents during July— their first month, and again during February in order to have objective data 
to determine promotion to the next year of training). 
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Implications 
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Just as the program director considers several educational alternatives and then selects the 
most appropriate approach for educating residents, the process should be the same when 
performing evaluation. Performing meta-evaluation is one aspect of program evaluation which 
should be considered whenever possible. The programs in this study did not practice the process 
of meta-evaluation. By performing meta-evaluation, the residency director not only considers 
who the stakeholders are, but also what the purpose of the evaluation is, what approaches will be 
used, what issues will be addressed, what political considerations should be taken into account, 
what standards will be used, and what resources are available. It also makes the director consider 
broader issues such as practicality, usefulness, and ethics. 

Program evaluation plans need to be developed and written, as do research proposals and 
findings. The problems or issues being addressed should be written, and the process or 
methodologies used should be described in detail. Justification for using theoretical evaluation 
models should be given. Reliability, validity, and objectivity should also be addressed. Findings 
and conclusions should not only be written down, but also distributed to the stakeholders in an 
appropriate format. 

The programs studied tended to use fewer data sources when evaluating areas or people 
other than residents. Assessment of residents consisted of data from many different sources such 
as videotapes, audiotapes, standardized exams, OSCEs, written evaluations from rotation 
attendings and nurses, self-evaluation, procedure documentation and competencies. Faculty, 
curriculum and programs should be assessed or evaluated in a similar manner, utilizing as many 
sources as are useful and feasible. 

Argyris (1984) states that “organizational learning is a process of detecting and correcting 

error” (p. 46). Residency programs can learn from the information provided from evaluations as 
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if 

well. Argyris (1984) describes what he refers to as single loop learning and double loop 
learning. Single loop learning can be compared to when a resident is videotaped in an out- 
patient setting at a specific time. The videotaping occurs, then it stops. Double loop learning 
takes this a step farther and asks why this particular resident is being videotaped, or what the 
purpose behind videotaping is. In other words, the underlying organizational policies and 
objectives are reviewed during double loop learning. Program evaluation should consider 
addressing the inputs, processes, outcomes, context and decision-making processes in order to 
achieve a more balanced evaluation. 

Residency programs could probably benefit from faculty development in the areas of 
evaluation and program evaluation. By utilizing a program evaluation template, and performing 
meta-evaluation, program directors will hopefully not only be able to obtain the “big picture” but 
also critically reflect on their programs. 
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